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(57) Abstract 

The invention provides a method of and a device for coding (10*) an image sequence (g(ij,k)). The device (10*) has a noise filter (12) 
for noise filtering (12) the image sequence (g(ij,k)) f and means (12) for regarding the noise filtering (12) as a rate-distortion optimization 
problem in order to adapt the response of the noise filter (12). In particular, a filter parameter set (C) is determined to adapt the response of 
the filter (12) in that the image sequence (g(ij,k)) is encoded using an optimal bit-budget, which is the bit-budget used for compressing a 
noise-free image sequence, and the distortion for the given bit-budget is minimized. A solution of the rate-distortion problem is proposed 
that calculates (12) the filter parameter set (C) by a Lagrange multiplier method. Separate minimization is used to determine each parameter 
of the parameter set (C) independently. In a practical embodiment, a spatial adaptive weighted averaging (SAW A) filtering (23) is used to 
estimate the bit budget (Rf) and to pre-filter the image sequence ig(ij,k)). 
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Coding and noise filtering an image sequence. 



The invention relates to a method of and a device for coding an image 
sequence, in which the image sequence is noise filtered. 

Kleihorst et al. [1] describe a single-chip MPEG2 encoder that employs an 
5 adaptive noise filter. The structure of the noise filter 2 is shown in Fig. 1. Besides the noise 
filter 2, the prior art embodiment has a subtractor 1, an adder 3, a selector 4, a main loop 
memory 5 and a motion estimator 6. For simplicity reasons, the signals are represented as 1- 
dimensional data. Here g(k) is the observed, noisy, signal; f MC {k — 1) is the previously filtered 
(and coded-decoded) signal which is retrieved from the main loop memory 5. Note that the 

10 subscript MC denotes that this signal is motion compensated to lie along the motion trajectory 
of g(k). The result of the filtering is /(*) , this is inserted into the coding chain instead of the 
noisy g(k). Within inter-coded macro-blocks, the signal / - f MC (k - 1) is forwarded. The 
Kalman-gain multiplier 0 < C < 1 is controlled to adapt the filter 2 to the situation at hand. 
Setting C = 1, forwards the (noisy) observation and no filtering action takes place. If C = 0, 

15 then only the prediction is forwarded, while for intermediate values of C noise filtering is 
achieved. Except from globally controlling this value, C is also directly controlled by the 
result of the motion compensation. This is done to avoid blurring in situations where MPEG's 
(translational-) motion model is insufficient. Reversely, the adaptivity also increases filtering 
effort if a good motion-compensation is detected. 

20 

An object of the invention is, inter alia, to provide a more effective filtering. To 
this end, the invention provides a method of and a device for coding an image sequence and a 
camera system as defined in the independent claims. Advantageous embodiments are defined 
in the dependent claims. 

25 A first embodiment of the invention is based on regarding the noise filtering as 

a rate-distortion optimization problem in order to adapt the noise filtering. The invention takes 
into account both the reduced quality of the encoded pictures and the increased amount of bits 
to transmit due to noise corruption. The invention allows adaptive filtering of image sequences 
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resulting in better compression and distortion performance. Preferably, a filter parameter set is 
determined to adapt the response of the filter in that: the image sequence is encoded using an 
optimal bit-budget (or rate) R which is used for compressing a noise-free image sequence, and 
the distortion D for the given bit-budget R is minimized. The distortion of a certain frame due 
5 to the encoding process and the number of bits used to encode that frame need to be evaluated. 
The rate-distortion problem can be solved efficiently by calculating the filter parameter set by 
a Lagrange multiplier method. 

A further embodiment according to the invention uses an effective algorithm to 
estimate an optimal Lagrange multiplier by determining a maximum of a second derivative of 

10 the bit budget R. In this embodiment the optimal Lagrange multiplier is determined without 
prior knowledge of the rate constraint and noise characteristics. 

A practical embodiment according to the invention uses spatial adaptive 
weighted averaging filtering to estimate the rate. In a further embodiment the spatial adaptive 
weighted averaging filtering is used to pre-filter the image sequence. This is done to exploit 

15 the spatial correlation between the pixels and to reduce the processing burden compared to 
three-dimensional filtering techniques. The displacement vectors are now estimated from the 
current and the previous frame after they are spatially filtered, which provides additional 
robustness to the motion estimates. 

The aforementioned and other aspects of the invention will be apparent from 

20 and elucidated with reference to the embodiments described hereinafter. 

In the drawings: 

Fig. 1 shows the structure of an adaptive noise filter as known from the prior art 
(already discussed); 

25 Fig. 2 shows a first embodiment according to the invention that provides 

combined video coding and noise reduction; 

Fig. 3 shows a second embodiment according to the invention that provides 
combined video coding and noise reduction; and 

Fig. 4 shows a camera system according to the invention. 
30 The drawings only show those elements that are necessary to understand the 



The invention regards noise filtering of an image sequence as a rate-distortion 
optimization problem in that the response of a filter can be adapted. In particular, an optimal 



invention. 




WO 00/42772 PCT/EP99/10414 

3 

filter parameter set is determined such that noisy images are encoded using an optimal bit 
budget and the distortion is minimized. The optimal bit budget is a number of bits that would 
be used for compressing a noise-free image sequence. As a favorable implementation, a 
solution based on a Lagrange multiplier method will be proposed. This method enables an 
5 efficient solution of the rate-distortion problem. Moreover, an effective algorithm will be 
discussed, which determines an optimal Lagrange multiplier without prior knowledge of the 
bit budget, i.e. a rate constraint, and noise characteristics. 

Generally speaking, perfect noise filtering cannot be achieved without 
degrading the original signal. An aspect of the invention is that the problem of filtering the 
10 image sequence in the context of a certain video coding is considered as follows: 

- encode the noisy sequence using the optimal bit budget R, i.e. the number of 
bits used to encode a noise-free sequence; 

- given R, minimize the distortion due to filtering in that the coefficients of a 
given noise filter are adapted. 

15 This can be regarded as a resource allocation problem where the optimal 

parameter set which allows efficient distribution of a given bit budget must be found in order 
to ensure satisfactory image quality. The invention, can be regarded as a general rate-distortion 
optimization framework to determine the optimal filter parameter set for a given video coder. 

Fig. 2 shows a device 10 according to the invention that shows combined video 

20 coding and noise filtering. The device 10 comprises a subtracter 1 1, a noise filter 12 with a 
processing unit CPU 12*, an adder 13, a switch 14, a Discrete Cosine Transformer (DCT) 15, 
a Quantizer (Q) 16, a variable length coder (VLC) 17, an inverse quantizer (Q* 1 ) 18, an inverse 
DCT (DCT 1 ) 19, an adder 20, a motion estimator (ME)/ motion compensator (MC) 21 and a 
switch 22. The device 10 is a hybrid coder that codes the incoming image sequence g(i,j,k) 

25 spatially and temporally by DCT transformation plus quantization and motion compensation. 
The device produces intra-coded frames (I) and inter-coded frames (P). Inter-coded frames are 
motion-compensated while intra-coded frames are not. If the frame should be intra-coded, the 
switches 14 and 22 are positioned such that no motion-compensation is performed. All frames 
are DCT coded in the DCT 15, quantized in Q 16 and entropy coded in VLC 17. To obtain a 

30 reconstructed frame, that is necessary for motion estimation and motion compensation, the 
coded frame is inverse quantized in Q* 1 18 and an inverse DCT transformed in DCT 1 19. A 
reconstructed frame is stored in a frame memory that is included in the ME/ MC unit 21. The 
noise filter 12 filters a differential frame, because it is coupled to the subtractor 11. The device 
10 and especially the noise filter 12 with the CPU 12* will be discussed in more detail below. 
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Assume an exemplary, simple additive noise model, given by: 
g(ij,k) = f(ij,k) + n(ij 9 k) (1) 
where g(ij,k) denotes the observed image sequence that is input to the subtractor 11 9 f[iJ,k) the 
5 original sequence and n(ij,k) the noise; ij are the spatial coordinates and k the discrete time 
variable (frame index). The noise n(ij,k) is assumed to be zero-mean, white, independent of 
f[ij,k) and Gaussian distributed with constant variance . 

As an illustration and in order to keep the computation low, the following 
discussion is focussed on a simplified motion compensated (MC) noise filter [2] producing a 
10 filtered frame: 
F(iJ,k) = 

F(i + v*, (*), j + v>, (*), k - 1) + C tJ (k)(g(i y j, k) - F(i + v*,. (*), j + vl (*), k - 1)) (2) 

wherein v uj {k) = [v* y (*) v'j(k)Y is the displacement of position (ij) and Q/k) is a control 
parameter to adapt a response of the noise filter 12. 

For the given MC video coding system 10, consider now an image that is 
15 partitioned into Kj x K2 fixed-sized block regions for motion estimation and motion 

compensation; the corresponding displacement set is d k = (d 0 0 (*),..., d Ki _ l K ^ x (Jfc)) . Besides, 

suppose that the image is partitioned into Nj x N 2 non-overlapping blocks of dimensions IxJ 
for MC filtering. The corresponding control parameter set and displacement set are given by: 

(*)) en \ k =(v 0O (*),..., 

(k)) respectively. 

20 The control parameter set and the motion vectors are assumed to be both in a finite set of 
admissible values. 



25 



F** <«'. h k) = F(i + v*„ 2 (*), j + v> >/l2 (k) y k - 1) 

+ C nM2 (k)\g nUn2 (1, j,*) - F(/ + v* . n2 (*), y + vj^ (*).* - 1)] 

denotes the filtered frame to be encoded, and 

F klJt2 (h j. k) = F(i + <f 4 * ua (*), j + rf/ U2 (*),* - 1) 

+ eki.„2 a. h k) - Fa + d* kU2 (fc), > + rf/ ua (*),*- d] 

the reconstructed filtered frame as displayed at a decoder wherein Q[.] is a 
quantization operator as present in quantizer 16. Note that kl and k2 are indices to indicate a 
block region for motion estimation and motion compensation. 
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Two separate motion estimation/ compensation processes have to be performed: 
one for the noise filtering and an other for the motion compensated coding. In order to save 
computational effort, it is assumed here that F(iJ t k) = g(ij,k) and that the sequence g(ij,k) 
being encoded is filtered along the motion trajectory v* = d* [1]. This means that the filtering is 
5 based on the motion compensated prediction performed in the coding loop. See Fig. 2. 

The reconstructed filtered frame is then given by: 
Fkija ft * *) = FQ + d; w (*), j + rf/ ua (*),* - 1) 

+ Qk,..2 Wfe„U2 ft h k) - F(i + d X kU2 (*), j + dl Uk2 (*), k - 1))] 

10 Notice that, since motion estimation is performed from the noisy observation 

g(ij,k), the noise may result in inaccurate motion estimates, which degrade the motion- 
compensated prediction. Therefore, a noise-robust motion estimator (ME) 21 for the 
computation of d* is adopted in an advantageous embodiment. A low-complexity spatio- 
temporal recursive ME [3] can be used. 

15 A rate-distortion optimization problem is formulated to compute the control 

parameter set C*. Let R[ be the number of bits used to encode a displaced frame difference 
(DFD) associated with the noise-free sequence f[ij t k), denoted as: 

DFDif. /) = Q(f kU2 (i, j\ k) - /(I + < U2 (*), j + d> w (*), k - 1)) (6) 
The goal is to find the set C* which minimizes the frame distortion: 
20 D* (C k ) = Function(g(i, j, k) y F(i, j 9 k)) for the given bit rate constraint R[ . 

Note that, since f[ij,k) is not available, only the distortion due to the filtering by 
means of £>*(Ca) can be taken into account but it is not possible to consider the degradation 
due to noise. 

Let AfunzCC* ) and R% Un2 (C k ) represent the distortion and the rate associated 
25 with the region (nj,n 2 ) respectively. Then the RD optimization problem can be expressed as: 
min D< (C, ) = imn £ D' nUn2 (C k ) (7) 

subject to R* = X (C k ) = R{ (8) 

n„rt 2 

Equations (5), (7) and (8) define the proposed RD optimized MC temporal noise filter for 
video coding. 
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In order to solve this problem efficiently, an unconstrained problem can be 
formulated by merging the rate term with the distortion term through a Lagrange multiplier 
X k [4]. Introducing the total Lagrange cost function: 

AC t ,A t )= 2b^(C t )+A t J?^(C t )J (9) 

5 It has been shown [5] that if there is a X k such that : 

C k =arg\mnJ(C k J k )\ (10) 
leadsto 3?/ = 5X<,(C*) = */ (11) 

n,.n 2 

then C k is also an optimal solution to (7), (8). 

10 A bisection method can be used to find an optimal X k . Supposing that the 

partition for noise filtering is such that there is no inter-region dependency, both rate arid 
distortion of the region (ni,n 2 ) depend only on the single C ni „ 2 (k) associated with that region. 
Thus, we have 

/C,< C *> = *.U<<W*>> (12) 
1 5 and D^ 2 (C k ) = (C n( , a (*)) (13) 

where the distortion is expressed as: 

(C„,. BJ (*)) = -^-ES ft M) - F n ^ (,-, j,k)f 



i_ _J_ 

(14) 

Therefore, the principle of separate minimization can be invoked, converting 
the joint optimization problem of (10) into a simpler optimization problem in which each 
20 control parameter (*) e can be determined independently, according to: 

C ni „ 2 (*) = arg c min jp^ (C„ + (C^ (*))J (15) 



Since in general there is no prior knowledge of the rate R( associated with the 
frame f(Uj,k), it would not be possible to compute X k such that (1 1) is fulfilled. An efficient 
25 method is proposed that is base on a scheme as described in [6]. This scheme implicitly 
estimates the noise power from the input data without the need for additional a priori 
information. An estimate A/ of the optimal X k is computed as: 
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A, = argmax * (16) 

and assumed is that J?/ (A/ ) = R* (17) 
An explanation of this equation is as follows: starting from X k = 0 a 

decreasingly smaller bit budget is allocated to encode the image sequence corrupted with 
5 additive noise, thus reducing resources to encode noise and increasing the compressibility of a 
picture. At a certain X k which results to be a knee-point, the exact amount of bits to encode 
the original sequence is allocated. After this value, the compression ratio increases slowly, 
which points out that fewer bits than necessary are allocated to encode video information. 

In a second embodiment 10* of the invention as shown in Fig. 3, a spatial 
10 adaptive weighted averaging (SAW A) filter 23 is used as an estimator of the rate term R{ . A 

detailed description of the SAWA filter can be found in [7]. The SAWA estimate h(i,j,k) at the 
pixel location (ij) in the kth frame is defined as 

h(i,j,k)= 2,yv(m,nJ)g(m,nJ) (18) 

where w(m, nj) = ■ _ K(iJ > k) (19) 

1 + a max , Lg(i, j , k) - g(m, n, l)f j 

15 are the weights within the spatial Support S(ij,k) 9 defined as the 3x3 spatial window centered 
about the current pixel location. K(iJ,k) is a normalization constant and a and e are tuning 
parameters. The quantity a, usually set at 1, controls how rapidly the weights should reduce as 
a function of the mismatch between pixels values, whereas the parameter € 2 determines the 
switching between weighted and direct averaging. In order to obtain accurate estimates of the 

20 number of bits to encode the DFD regardless of the noise level, a relationship between the 

parameters 2 and the noise variance cr 2 may be determined experimentally. From a number of 
experiments has been achieved that e 2 = (So 2 ) 2 , where 8 is a tuning constant. The estimate 
R[* of the rate R[ is then the number of bits used to encode: 

DFD{K h) = G(V*2 0\ h *) " * 0" + 4?u2 (*)■ J + * - 1)) (20) 

25 The number of bits R(* is determined in VLC 17 and furnished to the CPU 12* of the noise 
filter 12. The parameter set can be computed for each frame, but also kept constant for a 
certain number of frames. 
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In general, three-dimensional filtering techniques may be employed to exploit 
both the spatial correlation between the pixels and the temporal correlation between the 
frames. Furthermore, in order to reduce the processing burden, the filtering procedure may be 
separated in a spatial part, which operates on each frame separately, and a temporal part 
5 operating in the direction of motion. This technique is more advantageous especially for low 
SNR's since the displacement vectors are now estimated from the current and the previous 
frame after they have been spatially filtered, which provides additional robustness to the 
motion estimates. According to this strategy, SAWA pre-filtering of the current frame g(i,j,k) 
may be done first, and then the MC temporal filter may be applied to the smoothed images; i.e. 
10 g(i,j,k) of equation (5) is replaced by h(i,j,k) of equation (18), see Fig. 3. In order to achieve 
optimal distortion performance, e 2 = 2a\ is set for SAWA pre-filtering. 

A video coder according to the invention can be used in many applications, 
such as broadcast, video-phone, videoconferencing systems, satellite observation, surveillance 
systems, etc. Fig. 4 shows a camera system according to the invention, comprising a camera 
15 100 and a device 200. The camera furnishes the image sequence g(ij f k) to the device 200. The 
device 200 is similar to the device 10 or the device 10* as shown in Figs. 2 and 3 respectively. 

The invention is especially applicable on low bit-rate motion compensated 
hybrid coding schemes, such as H.261 and H.263, but also on MPEG. 

In summary, the invention provides a method of and a device for coding an 
20 image sequence g(i,j,k). The device has a noise filter for noise filtering the image sequence 

g(i,j,k) y and means for regarding the noise filtering as a rate-distortion optimization problem in 
order to adapt the response of the noise filter. In particular, a filter parameter set C is 
determined to adapt the response of the filter in that the image sequence g(i,j,k) is encoded 
using an optimal bit-budget, which is the bit-budget used for compressing a noise-free image 
25 sequence, and the distortion for the given bit-budget is minimized. 

A solution of the rate-distortion problem is proposed that calculates the filter 
parameter set C by a Lagrange multiplier method. Separate minimization is used to determine 
each parameter of the parameter set C independently. In a practical embodiment, spatial 
adaptive weighted averaging filtering is used to estimate the bit budget and to pre-filter the 
30 image sequence g( i,j,k). 

It should be noted that the above-mentioned embodiments illustrate rather than 
limit the invention, and that those skilled in the art will be able to design many alternative 
embodiments without departing from the scope of the appended claims. In the claims, any 
reference signs placed between parentheses shall not be construed as limiting the claim. The 
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word "comprising" does not exclude the presence of other elements or steps than those listed 
in a claim. The invention can be implemented by means of hardware comprising several 
distinct elements, and by means of a suitably programmed computer. In a device claim 
enumerating several means, several of these means can be embodied by one and the same item 
5 of hardware. 
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CLAIMS: 



1. A method of coding (10,10*) an image sequence (g(i,j,k)\ the method 
comprising the steps of: 

noise filtering (12) the image sequence (g(ij,k)); and 
adapting (12*) said noise filtering (12) in dependence on a rate-distortion 
5 optimization of said coding (10,10*). 

2. A method as claimed in claim 1, wherein the adapting step (12*) comprises the 
steps of: 

calculating (12*) a filter parameter set (C) of said noise filtering (12) by a 
10 Lagrange multiplier method; and 

estimating (12*) an optimal Lagrange multiplier by determining a maximum of 
a second derivative of a rate used in the rate-distortion optimization. 

3. A method as claimed in claim 1, wherein the adapting step comprises the steps 
15 of: 

spatial filtering (23) the image sequence; and 

taking (17) a number of bits that is used to encode the spatially filtered image 
sequence as a rate used in the rate-distortion optimization. 

20 4. A device (10,10*) for coding an image sequence (g(ij,k)) f the device 

comprising: 

means (12) for noise filtering the image sequence (g(i,j,k)); and 
means (12*) for adapting a response of said means for noise filtering (12) in 
dependence on a rate-distortion optimization of said coding (10,10*). 

25 



5. A camera system comprising a camera (100) and a device (200) as claimed in 

claim 4. 
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